Combination ultrasound / optical  image for an image-guided procedure

ABSTRACT

A system includes an image guidance system with a memory with computer executable instruction, a processor configured to execute the computer executable instructions, and a display. The computer executable instructions cause the processor to: receive a three-dimensional model of vasculature from an ultrasound imaging system, receive a real-time optical feed of an interior of a cavity from an optical camera-based guidance system, receive a first tracking signal indicative of a first spatial location of a probe of the ultrasound imaging system, receive a second spatial location of the optical camera-based guidance system, and overlay the optical feed with the three-dimensional model based on the first and second tracking signals. The display is configured to visually present the optical feed with the three-dimensional model overlaid thereover.

TECHNICAL FIELD

The following generally relates to image guidance and more particularlyto a combination ultrasound/optical image for an image guided procedure.

BACKGROUND

An image-guided procedure includes a procedure within a cavity of anobject where the clinician cannot see into the cavity through the objectand instead uses a displayed image of the inside of the cavity as aguide to maneuver and employ an instrument in the cavity. In oneexample, this includes a surgeon operating an optical camera-basedguidance system while performing the procedure. The procedure could be aresection, cauterization, cryotherapy, biopsy, ablation, etc. Theoptical camera-based guidance system can be a laparoscope inlaparoscopy, a microscope in neuro surgery, an optical interface of asurgical robot, etc.

With such a procedure, the surgeon can only see the surface of thetissue to be treated and not the tissue below the surface of the tissueto be treated. In order to avoid inadvertently damaging other tissue,e.g., cutting a vessel just below the surface of the tissue beingtreated, the surgeon has relied on a pre-operative image (e.g., computedtomography (CT) or magnetic resonance (MRI)) and/or a real-time image(e.g., ultrasound (US)). Unfortunately, a pre-operative image is limitedin that it does not represent a current state of the tissue, which mayhave shifted or deformed since the imaging, and it is difficult toconcurrently treat the tissue while scanning with the ultrasound probe.

SUMMARY

Aspects of the application address the above matters, and others.

In one aspect, a system includes an image guidance system with a memorywith computer executable instruction, a processor configured to executethe computer executable instructions, and a display. The computerexecutable instructions cause the processor to: receive athree-dimensional model of vasculature from an ultrasound imagingsystem, receive a real-time optical feed of an interior of a cavity froman optical camera-based guidance system, receive a first tracking signalindicative of a first spatial location of a probe of the ultrasoundimaging system, receive a second spatial location of the opticalcamera-based guidance system, and overlay the optical feed with thethree-dimensional model based on the first and second tracking signals.The display is configured to visually present the optical feed with thethree-dimensional model overlaid thereover.

In another aspect, a method includes receiving a three-dimensional modelof vasculature from an ultrasound imaging system. The method furtherincludes receiving a real-time optical feed of an interior of a cavityfrom an optical camera-based guidance system. The method furtherincludes receiving a first tracking signal indicative of a first spatiallocation of a probe of the ultrasound imaging system. The method furtherincludes receiving a second spatial location of the optical camera-basedguidance system. The method further includes overlaying the optical feedwith the three-dimensional model based on the first and second trackingsignals. The method further includes displaying the optical feed withthe three-dimensional model overlaid thereover.

In another aspect, a computer readable medium is encoded with computerexecutable instructions. The computer executable instructions, whenexecuted by a processor of a computer, cause the processor to: receive athree-dimensional model of vasculature from an ultrasound imagingsystem, receive a real-time optical feed of an interior of a cavity froman optical camera-based guidance system, receive a first tracking signalindicative of a first spatial location of a probe of the ultrasoundimaging system, receive a second spatial location of the opticalcamera-based guidance system, overlay the optical feed with thethree-dimensional model based on the first and second tracking signals,and display the optical feed with the three-dimensional model overlaidthereover.

Those skilled in the art will recognize still other aspects of thepresent application upon reading and understanding the attacheddescription.

BRIEF DESCRIPTION OF THE DRAWINGS

The application is illustrated by way of example and not limited by thefigures of the accompanying drawings, in which like references indicatesimilar elements and in which:

FIG. 1 schematically illustrates an example ultrasound system with acolor flow mapping processor;

FIG. 2 schematically illustrates a non-limiting examples of thesegmentation processor of the ultrasound system.

FIG. 3 shows an example color flow image;

FIG. 4 shows an example Power-Doppler image; and

FIG. 5 illustrates an example method in accordance with an embodimentherein.

DETAILED DESCRIPTION

The following general describes an approach to visualizing vasculatureduring an image-guided procedure of tissue. In one instance, thisincludes sweeping an array of an ultrasound transducer over the tissuewhile acquiring at least flow data (e.g., color flow mode (CFM),Doppler, etc.), creating a three-dimensional (3-D) volume with thescanned image plains, segmenting vasculature from the 3-D volume basedon the flow data to create a 3-D model of the vasculature, anddisplaying the 3-D model overlaid over an optical image of an opticalcamera-based guidance system while performing the procedure. The 3-Dmodel, in one instance, provides image-guidance on location ofvasculature so that the surgeon can avoid/minimize damaging thevasculature.

FIG. 1 illustrates a system 100 including ultrasound imaging system 102,an optical camera-based guidance system 104, a tracking system 106, animage-guidance system 108, and at least one instrument 110 configuredfor resection, cauterization, cryotherapy, biopsy, ablation, etc.

The ultrasound imaging system 102 includes a probe 112 and an ultrasoundconsole 114, which interface through suitable complementary hardware(e.g., a cable 113, as shown) and/or wireless interfaces. The probe 112includes a transducer array 116 with one or more transducer elements118. The transducer array 116 can be one- or two-dimensional, linear,curved, and/or otherwise shaped, fully populated, sparse and/or acombination thereof, etc. In one instance, the transducer array 116 isconfigured to mechanically and/or electrically rotate to capture 3-Ddata.

The one or more transducer elements 118 are configured to convert anexcitation electrical signal to an ultrasound pressure field. The one ormore transducer elements 118 are also configured to convert a receivedultrasound pressure field (an echo) into an electrical (e.g., analogradio frequency, RF) signal. The ultrasound pressure field, in oneinstance, is produced in response to a transmitted ultrasound pressurefield interacting with structure, such as blood cells flowing in aportion of a vessel and/or other tissue, and/or other tissue.

The probe 112 further includes a probe tracking device 120, whichinclude one or more tracking elements. One or more of the trackingelements can be external to the device 120 and/or one or more of thetracking elements can be internal to the device 120. In one instance,the one or more of the tracking elements include one or more of anemitter, a transmitter, and/or a passive sensor. Examples of suchtracking devices include an electro-magnetic tracking device, an opticaltracking device, an inertial tracking device, and/or other trackingdevice. Tracking devices are discussed in Birkfellner et al., “TrackingDevices,” In: Peters T., Cleary K. (eds) Image-Guided Interventions.Springer, Boston, Mass., 2008.

Transmit and receive circuitry (TX/RX) 122 is configured to generate theexcitation signal conveyed to the transducer array 116 for at least flowimaging, including 3-D imaging by manual and/or electrical-mechanicalsweeping of the transducer array 116. The TX/RX 122 is also configuredto process the electrical signal corresponding to the received echosignal. The TX/RX 122, in one instance, is further configured topre-condition and/or pre-process the signal (e.g., amplify, digitize,etc.). Other processing is also contemplated herein.

The illustrated embodiment shows the transmit and receive operations areperformed by the same circuitry, the TX/RX 122. In a variation, thetransmit and receive operations are performed by separate circuitry,e.g., transmit circuitry for transmit operations and separate receivecircuitry for receive operations. One or more switches and/or otherdevice(s) can be used to switch between transmit and receive operationsand/or transmit and receive circuitry by electrically connecting andelectrically disconnecting transmit and receive circuitry.

A beamformer 124 beamforms the signal, e.g., via delay-and-sumbeamforming and/or other beamforming. The beamformer 124 outputs thebeamformed data. An image processor 126 processes the beamformed data.For B-mode imaging, this includes generating a sequence of focused,coherent echo samples along focused scanlines of a scanplane. The imageprocessor 126 can also be configured to generate an A-mode, C-mode,and/or other ultrasound imaging mode image.

A flow processor 128 processes the beamformed data and generates flowimages. Suitable flow processing includes color flow, Doppler and/orother flow processing. Generally, color flow is an enhanced form ofDoppler that uses color to highlight the direction of blood flow. Insome embodiments, the flow processor 128 is also configured to detectpresence of flow (i.e. flow or no flow per voxel), e.g., using high passfiltering, singular value decomposition, principal component analysis,spatial coherence factors, etc.

Example approaches are discussed in Birkeland, et al., “Doppler-based 3DBlood Flow Imaging and Visualization,” Proceeding SCCG '13 Proceedingsof the 29th Spring Conference on Computer Graphics, Pages 115-122, May1-3, 2013, Torp, “Clutter Rejection Filters in Color Flow Imaging: ATheoretical Approach,” IEEE Transactions on Ultrasonics, Ferroelectrics,and Frequency Control, Vol. 44, No. 2, March 1997, and Demene et al.,“Spatiotemporal Clutter Filtering of Ultrafast Ultrasound Data HighlyIncreases Doppler and fUltrasound Sensitivity,” IEEE Transactions onMedical Imaging, Vol. 34, No. 11, November 2015.

A segmentation processor 130 is configured to segment the beamformeddata based on the flow data. For example, in one instance, thesegmentation processor 130, for each scan plane, distinguishes flow fromsurrounding tissue, and generates 3-D model representing vasculaturetissue (i.e. blood vessels). In one instance, this includes comparingthe data in each scan plane to a predetermined threshold and classifyingregions satisfying the predetermined threshold as flow data andclassifying other regions as non-flow data. The regions classified asflow data is then segmented and used to generate the 3-D model. In someembodiments, the segmentation processor 130 utilizes the B-mode imagesto refine the 3-D model, e.g., at border the between the vasculaturetissue and the surrounding tissue.

FIG. 2 schematically illustrates a non-limiting example of thesegmentation processor 130. Inputs to the segmentation processor 130include tracking data 204, a flow image 206, and a tissue-flow mask 208.The tracking data 204 come from the tracking device 120 and trackingsystem 106. The flow image 206 and the tissue-flow mask 208 are producedby the flow processor 128. The flow image 206 can be a color-flow imageor a power doppler image. The tissue-flow mask 208 can be a separatedata stream or incorporated in the flow image 206. The tissue flow-maskconsists typically of values between 0 and 1 indicating the probabilitythat a given pixel of the flow image is true blood flow (value of 1) ornot (value of 0).

Briefly turning to FIGS. 3 and 4, examples of a color flow image and apower doppler image are shown. FIG. 3 shows a 2D ultrasound image 302consisting of a B-mode image 304 overlaid with a color flow image 306.The display also includes a standard depth axis 308 and color-bars 310and 312 for the B-mode image 304 and the color flow image 306,respectively. FIG. 3 further shows labels 314 for predeterminedanatomical structures, which in the illustrated case are: umbilical vein(UV), stomach (S), portal vein (PV), and gallbladder. FIG. 4 shows ascreen-shot 402 consisting of a 3D-rendered Power-Doppler image 404, thedepth axis 308, the color-bar 310 for the B-mode image, and a color bar406 for the Power-Doppler image 404. The overlay includes labels 408 forpredetermined anatomical structures, which in this case are: aorta;inferior vena cava (ivc); ductus venosus (dv); and umbilical vein (uv).The user can rotate, translate and zoom the vessels rendered in 3D.

Returning to FIG. 2, The flow image 206 and the tissue-flow mask 208 canbe 2D planar images, or 3D volumetric scans, depending on the probe 112.In all cases, the tracking data 204 include coordinates and orientationof the origin of the images. The volume creation block 210 maps the flowimage 206 to a volumetric image. The created volume can be displayeddirectly to the user by the US display 132 by employing techniques suchas volume rendering or maximum intensity projection. An example of thisis discussed in Fishman et al., “Volume Rendering versus MaximumIntensity Projection in CT Angiography: What Works Best, When, and Why,”Radiographics, vol. 26, no. 3, pp. 905-922, 2006. In this case, theblocks 212 and 214 are bypassed by employing data paths 216 and 218.

The segmentation and labeling block 212 clusters the voxels that belongto the same vessel and assigns them labels. An example of this isdescribed in Bradski et al., “Learning OpenCV. Computer Vision with theOpenCV Library”, ISBN 9780596156022, O'Reilly Media, 2008. The output ofthe segmentation and labeling block 212 can be sent to the US display132. The surface generation block 214 uses the labeled voxel data tocreate triangle meshes of the surface of the vessel walls. Examples ofsuitable approaches are discussed in Treece et al, “Regularized marchingtetrahedra: Improved iso-surface extraction,” Comput. Graph., vol. 23,no. 4, pp. 583-598, 1999, Treece et al., “Fast surface and volumeestimation from non-parallel cross-sections, for freehandthree-dimensional ultrasound,” Med. Image Anal., vol. 3, no. 2, pp.141-173, 1999, and Treece et al., “Surface interpolation from sparsecross sections using region correspondence,” IEEE Trans. Med. Imaging,vol. 19, no. 11, pp. 1106-1114, 2000.

A display (US Display) 132 is configured to display images, e.g.,B-mode, flow images, and/or the 3-D model. A controller 134 isconfigured to control one or more of the components of the ultrasoundimaging system 100. Such control can be based on available modes ofoperation such as B-mode and/or flow mode, etc. A user interface 136includes one or more input devices such as a keyboard, a trackball, amouse, a touch sensitive screen, etc. configured to allow a user tointeract with the ultrasound imaging system 102.

In this example, the optical camera-based guidance system 104 includes alaparoscope 133, which includes a shaft 136, a camera 138 disposed at afirst end of the shaft 136, a light guide 140 disposed at the first endof the shaft 136, a light source 140 in optical communication with thelight guide 140, and a laparoscope tracking device 142, which includeone or more tracking elements. Other suitable systems include amicroscope, an optical interface of a surgical robot, etc. One or moreof the tracking elements can be external to the tracking device 142and/or one or more of the tracking elements can be internal to thetracking device 142. In one instance, the one or more of the trackingelements include one or more of an emitter, a receiver, and/or a passivesensor. Examples of such tracking devices include an electro-magnetictracking device, an optical tracking device, an inertial trackingdevice, and/or other tracking device. Tracking devise are discussed inBirkfellner et al.

The tracking system 106 interacts with the probe tracking device 120 andthe laparoscope tracking device 142, registers their spatial coordinatesystems, and determines a location and an orientation of the probetracking device 120 and the laparoscope tracking device 142 relative toeach other. For example, where the probe tracking device 120 and/or thelaparoscope tracking device 142 includes magnets, the tracking system106 measures a magnetic field strength of the magnets, which depends ona distance and direction of the magnets to the tracking system 106, andthe strength and direction is used to determine location andorientation. In another example, where the probe tracking device 120and/or the laparoscope tracking device 142 includes external opticalelements, the tracking system 106 includes an optical device such as avideo camera the records the spatial orientation of the optical elementsto determine location and orientation.

Examples of suitable tracking system 106 are described in U.S. patentapplication US 2010/0298712 A1, filed Feb. 10, 2010, and entitled“Ultrasound Systems Incorporating Position Sensors and AssociatedMethod,” and U.S. Pat. No. 8,556,815 B2, filed May 6, 2010, and entitled“Freehand Ultrasound Imaging Systems and Methods for Guiding ElongateInstruments,” both of which are incorporated herein by reference intheir entireties. Another example is discussed in U.S. Pat. No.7,835,785 B2, filed Oct. 4, 2005, and entitled “DC Magnetic-BasedPosition and Orientation Monitoring system for Tracking MedicalInstruments.” Other tracking systems are discussed in Birkfellner et al.Other approaches are also contemplated herein.

The image guidance system 108 includes at least a processor 144 (e.g., acentral processing unit, a microprocessor, etc.) and memory 146 (e.g.,physical memory, etc.). The image guidance system 108 receives the 3-Dmodel from the console 114 of the ultrasound imaging system 102, theoptical signal/feed from the optical camera-based guidance system 104,and the tracking signal from the tracking system 106, and displays, viaa display (IGS Display) 148, the optical signal/feed from the opticalcamera-based guidance system 104 overlaid with the 3-D model based onthe tracking signal. In one instance, the memory 146 include anaugmented reality algorithm, the processor 144 executes the augmentedreality algorithm to combine the optical signal/feed and the 3-D model.Other approaches are also contemplated herein. Depth information can beadded through shading, fading, intensity, color, etc.

The at least one instrument 110 includes a shaft 150 and a device 152disposed at a first end of the shaft 150. The device 152 can include asuitable device for resection, cauterization, cryotherapy, biopsy,ablation, etc. For example, the device 152 can include a grasper,scissors, a stapler, etc.

The probe 112 of the ultrasound imaging system 102, the opticalcamera-based guidance system 104, and the at least one instrument 110are shown in connection with an object 154 and material of interest 156within the object 154. Where the object 154 is a patient, a portal suchas a trocar or the like may first be placed through the wall of theobject 154 and the optical camera-based guidance system 104 and the atleast one instrument 110 are inserted into cannulas of the trocars andinto the cavity of the object 154.

FIG. 5 illustrates an example method in accordance with an embodimentherein.

It is to be understood that the following acts are provided forexplanatory purposes and are not limiting. As such, one or more of theacts may be omitted, one or more acts may be added, one or more acts mayoccur in a different order (including simultaneously with another act),etc.

At 502, the transducer array 116 is swept over the material of interest156.

At 504, the flow processor 128 determined flow information from theacquired data.

At 506, the segmentation processor 130 segments vasculature from theacquired data with the flow information.

At 508, the segmentation processor 130 creates the 3-D model of thevasculature with the segmentation.

At 510, the image guidance system 108 receives an optical feed from theoptical camera-based guidance system 104.

At 512, the image guidance system 108 receives the tracking signal fromthe tracking system 106.

At 514, the image guidance system 108 displays the optical feed with the3-D model projected thereon based on the tracking signal, which alignstheir coordinate systems.

This process can be repeated one or more times during the procedureand/or after the procedure, e.g., to validate vasculature was avoided.

The 3-D model can be static and outline the average contours of thevasculature or dynamic and show pulsation of the flow.

Depth information can be added through shading, fading, intensity,color, etc. The user utilizes the displayed information to guide theinstrument 110 to the material of interest 156 and treat the material ofinterest 156 with the instrument 110.

The above may be implemented by way of computer readable instructions,encoded or embedded on computer readable storage medium (which excludestransitory medium), which, when executed by a computer processor(s)(e.g., central processing unit (CPU), microprocessor, etc.), cause theprocessor(s) to carry out acts described herein. Additionally, oralternatively, at least one of the computer readable instructions iscarried by a signal, carrier wave or other transitory medium, which isnot computer readable storage medium.

The application has been described with reference to variousembodiments. Modifications and alterations will occur to others uponreading the application. It is intended that the invention be construedas including all such modifications and alterations, including insofaras they come within the scope of the appended claims and the equivalentsthereof.

What is claimed is:
 1. A system, comprising: an image guidance system,including: a memory with computer executable instructions; a processorconfigured to execute the computer executable instructions, which causesthe processor to: receive a three-dimensional model of vasculature froman ultrasound imaging system; receive a real-time optical feed of aninterior of a cavity from an optical camera-based guidance system;receive a first tracking signal indicative of a first spatial locationof a probe of the ultrasound imaging system; receive a second spatiallocation of the optical camera-based guidance system; and overlay theoptical feed with the three-dimensional model based on the first andsecond signals; and a display configured to visually present the opticalfeed with the three-dimensional model overlaid thereover.
 2. The systemof claim 1, wherein the three-dimensional model is a static model thatoutlines average contours of the vasculature.
 3. The system of claim 1,wherein the three-dimensional model is a dynamic model that visuallypresents pulsation of a flow.
 4. The system of claim 1, wherein theultrasound imaging system comprises: a probe with a transducer array toacquire three-dimensional data of the vasculature; a beamformerconfigured to beamform the acquired data; a flow processor configured toproduce flow data from the beamformed data; and a segmentation processorconfigured to produce the three-dimensional model from the flow data. 5.The system of claim 4, the ultrasound imaging system further comprises:an image processor configured to produce a B-mode image from thebeamformed data, wherein the segmentation processor refines thethree-dimensional model with the B-mode image.
 6. The system of claim 4,further comprising: a tracking device of the probe; and a trackingsystem configured to interact with the tracking device to produce thefirst tracking signal.
 7. The system of claim 1, wherein the opticalcamera-based guidance system comprises: a shaft; a light disposed at afirst end; an optical device disposed at the first end; a light sourcein optical communication with the light; and an optical path from theoptical device to the processor.
 8. The system of claim 7, furthercomprising: a tracking device of the optical camera-based guidancesystem; and a tracking system configured to interact with the trackingdevice to produce the second tracking signal.
 9. The system of claim 1,wherein the optical feed shows a location of an instrument in thecavity.
 10. The system of claim 1, wherein the three-dimensional modelshows vasculature behind an outer surface tissue being treated by aninstrument in the cavity.
 11. A method, comprising: receiving athree-dimensional model of vasculature from an ultrasound imagingsystem; receiving a real-time optical feed of an interior of a cavityfrom an optical camera-based guidance system; receiving a first trackingsignal indicative of a first spatial location of a probe of theultrasound imaging system; receiving a second spatial location of theoptical camera-based guidance system; and overlaying the optical feedwith the three-dimensional model based on the first and second trackingsignals; and displaying the optical feed with the three-dimensionalmodel overlaid thereover.
 12. The method of claim 11, wherein thethree-dimensional model is a static model that outlines average contoursof the vasculature.
 13. The method of claim 11, wherein thethree-dimensional model is a dynamic model that visually presentspulsation of the flow.
 14. The method of claim 11, further comprising:acquiring three-dimensional data of the vasculature; beamforming theacquired data; producing flow data from the beamformed data; andsegmenting the three-dimensional model from the flow data.
 15. Themethod of claim 11, wherein the optical feed shows a location of aninstrument in the cavity, and the three-dimensional model showsvasculature behind an outer surface tissue being treated by theinstrument in the cavity.
 16. A computer readable medium encoded withcomputer executable instructions which when executed by a processor of acomputer causes the processor to: receive a three-dimensional model ofvasculature from an ultrasound imaging system; receive a real-timeoptical feed of an interior of a cavity from an optical camera-basedguidance system; receive a first tracking signal indicative of a firstspatial location of a probe of the ultrasound imaging system; receive asecond spatial location of the optical camera-based guidance system;overlay the optical feed with the three-dimensional model based on thefirst and second tracking signals; and display the optical feed with thethree-dimensional model overlaid thereover.
 17. The computer readablemedium of claim 16, wherein the three-dimensional model is a staticmodel that outlines average contours of the vasculature.
 18. Thecomputer readable medium of claim 16, wherein the three-dimensionalmodel is a dynamic model that visually presents pulsation of the flow.19. The computer readable medium of claim 16, wherein the optical feedshows a location of an instrument in the cavity.
 20. The computerreadable medium of claim 16, wherein the three-dimensional model showsvasculature behind an outer surface tissue being treated by theinstrument in the cavity.